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An  Important  Issue  In  quantal  response  estimation  problems  considered 
by  Mazzuchl  and  Slngpurwalla  (1082)  Is  the  design  of  the  experiment.  The 
objective  there  was  to  estimate  the  probability  of  response  for  a given 
stimulus,  but  due  to  the  expense  of  the  Items,  testing  has  to  be  kept  to  a 

minimum.  As  a continuation  of  the  work  by  Mazzuchl  and  Slngpurwalla 

N< 

(1982),  we  address^thls  problem  and  presents  a criterion  for  comparing 
several  designs  that  are  of  Interest. 
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THE  DESICN  OF  A QUANTAL  RESPONSE  EXPERIMENT: 
AN  EMPIRICAL  APPROACH 


Refik  Soyer 


1.  INTRODUCTION 

The  U.S.  Army  Kinetic  Energy  Penetrator  problem  has  been  de- 
scribed by  Mazzuchi  and  Singpurwalla  (1982),  henceforth  MS.  Their 
objective  was  to  estimate  the  relationship  between  the  striking  velocity 
(the  stimulus)  and  the  probability  of  penetration  of  a projectile.  This 
is  a quantal  response  experiment  in  which  the  goal  is  to  estimate  the 
probability  of  response  for  a given  stimulus. 

The  strategy  used  to  test  the  effectiveness  of  the  penetrator  is 
to  fix  an  angle  of  fire  and  then  to  fire  the  penetrator  at  different 
striking  velocities.  After  each  firing,  the  outcome,  success  or  failure, 
is  recorded. 

The  equipment  used  in  testing  is  expensive,  and  thus  testing  is 
kept  to  a minimum.  Typically,  an  experimenter  is  allowed  a fixed  number 
of  tests.  That  is,  a fixed  number  of  copies  of  the  penetrator  can  be 
tested  at  different  striking  velocities.  Therefore,  designing  the  ex- 
periment in  an  optimal  way  is  an  important  issue.  In  a quantal  response 
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2 . 1 Simulation  of  the  Responses 


The  outcome  of  a test  at  is  described  by  a binary  variable 

X^  , i ■ l,2,...,k  , where  ■ 1 If  the  target  is  defeated  and  - 
otherwise.  To  simulate  the  outcome  X^  of  a test  at  stimulus  , we 
assume  that  we  know  the  "true"  probability  of  response  at  V ^ , 
i - 1,2, ... ,k  . 

Let  V,  < V.  < . . . < V.  be  the  selected  levels  of  stimulus  for 
12  k 

the  experiment;  then  (V^ , V^,  ...,  V^)  is  the  selected  design.  Let 
R(v)  be  the  "true"  response  curve;  the  response  curves  considered  here 
are  cumulative  distribution  functions.  Thus,  the  true  probability  of 
response  p^  at  stimulus  is  R(vp  . Next  we  generate  a random 

variable,  , from  a uniform  distribution  over  (0,1)  and  set  = 1 

if  < p^  , and  X^  - 0 if  > p^  . Thus  the  outcome  for  a given 

design  is  a k-dimensional  vector  of  0's  and  l’s. 

Once  X = (X^ , ....  X^)  is  obtained,  the  probabilities  of 

response,  p^'s  , i = l,...,k  , can  be  estimated  using  the  approach 
discussed  in  MS. 


2.2  Estimation  of  V g,. 

To  estimate  the  probability  of  response,  p^  , for  each  V.  , 

i * 1,2 k , we  assign  a Dirichlet  as  a prior  distribution  for  the 

successive  differences  p,  , p„-p, , ....  p, -p,  . and  the  modal  value  of 

rl  2 1 k k-1 

the  joint  posterior  distribution  is  a Bayes  point  estimate  of 
(p^,...,p^)  . The  computation  of  the  modal  value  of  the  joint  posterior 
distribution  necessitates  the  use  of  an  optimization  algorithm;  this  is 
described  by  Mazzuchi  and  Soyer  (1982). 


Thu  specification  of  the  prior  parameters  of  the  Dirichlet 


distribution  is  also  discussed  in  MS. 

Once  estimates  of  the  p^'s  are  obtained,  an  estimate  of  V ^ 
can  be  obtained  by  constructing  an  estimated  response  curve.  The  esti- 
mated response  curve  is  a plot  of  the  levels  of  stimulus  , versus 

A 

the  p^  s , the  estimated  probabilities  of  response,  i ■ l,2,...,k  . 

Once  such  a plot  is  obtained,  the  interpolation  procedure  described  in 
MS  is  used  to  estimate  V . 

Specifically,  for  the  estimation  of  ^ q5  ’ we  ^rst  see  ^ there 
is  an  observed  stimulus,  , for  which  p^  = 0.05  . If  so,  then 

is  the  estimate  of  ^ q5  * ^ not,  the  pair  of  observational  stimuli, 
say  and  , for  which  p^  < 0.05  < P^+^  » are  determined. 

Since  the  response  curve  is  increasing,  the  straight  line  segment 
joining  the  points  0,  p^,  ...,  p^  p^,  ...,  p^,  1 , will  be  an  in- 
creasing function  of  i . We  can  find  the  value  of  the  stimulus,  say 
^05  ’ Vi  ^ ^ 05  ^ Vi+1  ’ ^°r  w*1*c*1  P = 0.05  (as  indicated  in  Figure  1). 
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2.3  Comparison  of  Designs 

The  goal  of  our  analysis  is  to  select  a design,  (V  ,...,V,  ) , 
which  will  provide  a "good"  estimate  of  V ^ . 

In  order  to  determine  an  optimal  choice  of  the  k distinct 
levels  of  stimulus,  we  consider  different  designs,  and  first  obtain  an 
estimate  of  V for  each  design.  Let  (v|,  ...,  V^)  denote  design 
j ; the  superscript  j indicates  a particular  design.  Once  a j is 
chosen,  we  obtain  = (x|,  ...,  X^)  using  the  approach  discussed  in 
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Figure  1.  Interpolation  procedure. 

Section  2.2.  Then  via  the  estimated  response  curve,  we  obtain  an  esti 
mate  of  ^ q5  ’ sa^  ^05  * Since  the  "true"  response  curve  is  assumed 
to  be  known,  the  estimate  V can  be  compared  with  the  true  value  of 

V 

.05  ' 

If  the  above  procedure  is  repeated  for  a different  design,  a 
different  response  curve  is  estimated.  The  various  estimated  response 
curves  provide  us  with  different  estimates  of  ^ q5  ’ anc*  we  neet*  to 
determine  which  of  the  designs  gives  us  estimates  which  are  closest  to 

V qj  . Note  that,  since  the  outcome  = (x|,  ....  X^)  for  design  j 
is  obtained  by  simulation,  different  replications  of  X^  can  be  ob- 


tained by  using  different  seeds  in  the  simulation. 


Let  N bo  the  number  of  replications  which  are  analyzed  for  de- 
sign J . For  each  replication  of  , a different  response  curve  is 

/N  j 

estimated  and  therefore  a different  estimate  of  ^ q5  ’ Sl1^  ^ q^(£)  • 
is  obtained.  Since  we  know  the  true  value  of  V , the  mean  squared 
error  (MSE)  for  design  j is  computed  as 

MSEJ  = l <VJ  (St)  - V r . 

4-1  ,U3  * 

The  MSE  for  each  design  can  be  obtained  and  a comparison  of  the 
MSE's  provides  us  with  a criterion  for  selecting  a good  design.  The 
design  with  the  minimum  MSE  is  a good  design  for  a known  response  curve, 
say  R^(v)  . It  is  possible  that  a design  which  Is  good  for  R^(v)  may 
not  be  good  for  R^(v)  * i t k . This  possibility  has  also  been  consid- 
ered in  our  analysis. 


3.  SUMMARY 


The  approach  we  presented  in  Section  2 is  applied  to  some  simu- 
lated data  in  the  next  section. 

Three  different  "true"  response  curves  are  selected.  These 
curves  are  chosen  in  such  a way  that  they  will  provide  us  with  different 
values  of  v q5  - 

The  first  response  curve  is  specified  via  a Weibull  distribution 
function, 


R^v^  = 1 


The  second  is  via  a lognormal  distribution  function, 
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whore  V ^ = 52 


0.33 

and  $ denotes  the  standard  normal  distribution  function. 

The  third  response  curve  considered  is  also  a lognormal  distribu- 


tion function,  which  gives  V nc.  = 10  ; that  is, 

» U J 


R3(v1)  = 4> 


logo  v£  - 3.3 


0.6 


Five  different  designs  are  selected  and  analyzed. 

Design  1 — the  k observations  are  distributed  evenly  over  the 
entire  interval  of  the  range  of  testing,  say  I . 

Design  2 — all  the  k observations  are  concentrated  on  the 
left-hand  half  of  I . 

Design  3 — all  the  k observations  are  concentrated  in  the 
center  of  I . 

Design  4 — all  the  k observations  are  concentrated  on  the 
right-hand  half  of  I . 

Design  5 — the  k observations  are  sequentially  obtained  in 
three  different  phases. 

The  value  of  k is  (ari  rarily)  chosen  as  12,  and  due  to  the 
expense  of  simulation,  ten  different  replications  of  are  considered. 

The  MSE's  for  each  design  based  on  the  ten  replications  are  com- 
puted, on  the  basis  of  which  it  is  felt  that  Design  3 is  a suitable  de- 
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4.  APPLICATION  TO  SOME  SIMULATED  DATA 


The  three  "true"  response  curves  discussed  in  Section  3 are 


analyzed  separately  in  this  section.  These  response  curves  are  illus- 


trated in  Figures  2,  3,  and  4.  A replication,  simulated  from  each  of 


these  response  curves,  is  presented  in  the  Appendix  for  illustration. 


We  assume  that  the  probability  of  a response  at  a striking  veloc- 


ity of  300  is  almost  1.  Thus  we  make  an  arbitrary  choice  for  our  best 


prior  guess  of  p^  , say  p*  , by  lotting  p*  - 1 - exp[-0.0307  V^]  . 


The  prior  parameters  are  chosen  as  described  in  MS.  In  our  analysis  the 


smoothing  parameter  is  chosen  as  8 ■ 10  . 


The  five  different  designs  presented  in  Section  3 will  be  used  in 


the  analysis.  In  the  first  four  designs,  the  penetrator  is  tested  in  a 


single  phase.  In  Design  1,  the  12  observations  are  taken  equally  spaced 


over  the  entire  range  of  testing,  (0,300).  In  Design  2 all  12  observa- 


tions are  taken  equally  spaced  on  the  left-hand  half  of  the  interval 


(0,150).  In  Designs  3 and  4 the  12  observations  are  taken  equally 


spaced  in  the  center,  and  on  the  right-hand  half  of  the  interval. 


respectively . 


The  sequential  design,  Design  5,  consists  of  three  phases.  In 


the  first  phase,  six  observations  are  taken  equally  spaced  over  the 


entire  range  of  testing.  Ten  different  replications  of  the  outcome 


vector,  X , are  examined  and  the  experimenter  tries  to  identify  two 


regions:  one  region  where  the  outcome  is  zero  and  another  where  the 


outcome  is  one  most  of  the  time.  Once  these  two  regions  are  determined. 


the  experimenter  has  knowledge  about  the  region  where  the  response  is 
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Figure  4.  Lognorm.il  response  curve  II. 


most  likely  to  change.  In  phase  l of  the  experiment  a new  response 
curve  is  also  estimated,  and  this  curve  provides  the  prior  values  of 
P^'s  for  the  second  phase.  In  the  second  phase,  three  observations  are 
taken,  equally  spaced  over  the  region  where  the  response  is  most  likely 
to  change,  as  is  indicated  by  phase  1 and  a new  response  curve  estimated 
based  on  these  observations  and  the  prior  (the  posterior  from  phase  1). 
In  the  final  phase  of  the  experiment,  the  remaining  three  observations 
are  taken  on  the  left-hand  end  of  our  best  guess  based  on  phase  2 and  a 
new  response  curve  is  estimated  using  these  observations  and  the  prior 
(the  posterior  from  phase  2).  The  estimate  of  V ^ is  obtained  by 
using  this  updated  response  curve. 

The  outcomes  of  the  five  different  designs  are  presented  in  the 
Appendix,  Tables  A.l  - A. 3. 
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A . 1 Analyst-  for  the  Weibull  Response  Curve 

The  first  response  curve  that  is  considered  is  a Weibull  distribu- 


,J 


tion  function  for  which  V ^ = 22  . The  outcome  vector,  XJ  for 


j = 1,...,A  , is  simulated  using  this  response  curve.  Ten  replications 
of  the  outcome  vector  are  obtained  for  each  design.  One  of  these  repli- 
cations is  presented  in  Table  A.l  of  the  Appendix.  The  procedure  that 
was  discussed  in  Section  2.2  is  adopted  and  the  estimates  of  V are 
obtained.  The  "true"  response  curve  and  the  estimated  response  curve 
are  plotted  in  Figures  5,  6,  7,  and  8 for  one  replication,  and  presented 


in  Table  A.l.  The  estimates  of  V are  obtained  from  these  figures. 

. U-; 

A 


For  Design  1,  the  estimate  of  V 


05 


is  obtained  as  = A from  the 


estimated  response  curve  in  Figure  5.  Similarly,  the  estimates  of  V 


.05 


^2 

for  Designs  2,  3,  and  A are  obtained  as  V nc  = A , vnc  = 10  , and 

.05  .U5 


V.05  = 16  * 


For  the  sequential  design,  the  response  curve  that  is  estimated 
in  the  first  phase  is  presented  in  Figure  9 for  the  replication  presented 
in  Table  A.l.  The  response  curves  estimated  in  phases  2 and  3 are 


plotted  in  Figures  10  and  11,  respectively.  The  estimate  of 


' .05 


is 


obtained  from  Figure  11  as  V ^ = 8 , 


Once  the  VJq,-(£)  values  are  obtained  for  i,  = 1,...,10  for 


Design  j , MSEJ  can  be  computed  as: 


MSEJ  - l 


10 


10 


£=1 


V.05(£)  - 22 


MSEJ's  for  the  Weibull  response  curve  are  presented  in  Table  1, 
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Table  1.  MSE's  for  the 
Weibull  Response  Curve 


Design 

MSE 

1 

360.1 

2 

320.2 

3 

126.2 

4 

36.0 

5 (sequential) 

196.8 

The  minimum  MSE  is  obtained  when  Design  4 is  selected;  that  is, 
when  all  k observations  are  concentrated  in  the  right-hand  half  of  the 
interval  of  the  range  of  testing.  The  second  lowest  MSE  is  obtained 
when  all  k observations  are  concentrated  in  the  center  of  the  interval 
of  the  range  of  testing. 


4.2  Analysis  for  the  Lognormal  Response  Curve  I 

The  second  response  curve  is  a lognormal  distribution  function 
where  V n,  ■ 52  . Again  ten  outcome  vectors,  X^'s  , are  simulated  for 

• Uj  w* 

A j 

each  design.  The  response  curves  are  constructed  and  VJ,s  are  ob- 
tained. The  estimated  response  curves  can  be  observed  from  Figures  12, 
13,  14,  and  15  for  Designs  1,  2,  3,  and  4,  respectively,  for  a single 

a1  ^2 

replication.  The  estimates  of  V q,.  are  V 05  = ^ » v 05  = * 

V 0^  - 11  , and  V = 18  from  the  corresponding  figures. 

For  the  sequential  design,  Design  5,  the  response  curves  esti- 
mated in  phases  1,  2,  and  3 are  plotted  in  Figures  16,  17,  and  18,  re- 
spectively.  The  estimate  of  V is  obtained  as  v 05  = ^ from  Figure 
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Design  2,  lognormal. 


Curve 
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Sequential  design,  phase  III,  lognormal 


Table  2.  MSE's  for  the 
Lognormal  Response  Curve  I 


Design 

MSE 

1 

2313.7 

2 

2209.0 

3 

1714.2 

4 

1142.8 

5 

1981.1 

The  MSE's  computed  for  the  lognormal  response  curve  are  presented 
ia  Table  2.  As  we  can  observe,  the  minimum  MSE  is  obtained  when  Design 
4 is  selected.  The  second  lowest  MSE  is  obtained  for  Design  3. 

4 . 3 Analysis  for  the  Lognormal  Response  Curve  II 

The  third  response  curve  is  also  a lognormal  distribution  func- 
tion, where  V ^ = 10  . The  outcome  vectors  are  simulated  and  the 

~ j 

response  curves  are  estimated  as  in  the  previous  sections.  The  V 
values  are  obtained  using  the  estimated  response  curves  for  each  design. 
The  estimated  response  curves  can  be  observed  from  Figures  19  — 22  for 
Designs  1,  2,  3,  and  4 for  a single  replication.  The  estimates  are  ob- 

/si  A 2 A 3 ^ / 

tained  as  V qj  = 2 , V qj  = 3 , V ^ = 10  , and  V ^ = 19  . 

For  Design  5,  the  estimated  response  curves  for  phases  1—3  are 
presented  in  Figures  23—25.  The  estimate  of  V q,.  is  obtained  as 
V5  = 3 for  the  sequential  design. 

• U D 


Table  3.  MSE's  for  Che 
Lognormal  Response 
Curve  II 


The  MSE's  for  the  second  lognormal  response  curve  are  presented 
in  Table  3.  We  can  observe  from  Table  3 that  the  minimum  MSE  is  ob- 
tained for  Design  3,  where  all  12  observations  are  concentrated  in  the 
center  of  the  interval  of  the  range  of  testing.  The  MSE  obtained  for 
Design  4 is  the  highest  among  them  all.  This  indicates  that  the  form  of 
the  "true"  response  curve  affects  the  results  significantly. 


5.  CONCLUSION 


The  application  of  our  approach  to  simulated  data  from  three 
types  of  response  curve  indicates  that  the  shape  of  the  "true"  response 
curve  is  a significant  factor  in  the  evaluation  of  the  estimate  of  V 
In  real  life,  the  "true"  response  curve  is  never  known;  therefore  the 
experimenter  should  select  his  design  based  on  his  prior  knowledge  of 
the  problem.  Depending  on  the  shape  of  the  "true"  response  curve  that  is 
unknown  to  us,  the  V n,  level  might  be  underestimated  or  overestimated. 


Sometimes  the  discrepancy  it;  no  large  lliuU  one  ends  up  with  u high  MSK 
for  a given  design,  which  is  undesirable.  The  results  obtained  in* 

Section  A indicate  that  Design  2,  where  ail  k observations  are  concen- 
trated in  the  left-hand  half  of  the  interval  of  the  range  of  testing, 
lias  u tendency  to  underestimate  V ^ On  the  other  hand,  Design  A has 
a tendency  to  overestimate  V ^ For  the  response  curve  that  is  con- 
sidered in  Suction  A. 3,  this  caused  u high  MSU  for  Design  A.  The  possi- 
bility of  large  discrepancies  for  these  two  designs  makos  thorn < undesir- 
able. The  results  of  Section  A also  indicato  that  Design  3,  where  all 
k observations  are  concentrated  in  the  center,  gives  better  estimates 
of  V ^ in  general.  The  discrepancies  due  to  ovorostiination  or  under- 
estimation are  not  large.  This  makes  Design  3 more  desirable  than  the 
others. 

However,  one  should  noto  that  there  is  the  difficulty  of  deter- 
mining the  region  where  the  k observations  will  bo  concuntratcd.  if 
the  experiment  hes  to  be  performed  in  a single  phase,  this  region  can 
be  determined  by  using  past  information  available  to  the  invostigutor . 
Another  possibility  is  to  choose  the  middle  portion  of  the  lntervul  of 
the  range  of  testing. 

On  the  basis  of  the  analysis  made,  we  can  conclude  that  u dosign 
where  the  observations  are  concentrated  in  a region  that  provides  thu 
experimenter  with  more  information  is  suitable  for  this  problem.  There- 
fore, iu  our  analysis' Design  3 is  suggested  as  a suitable  design  for  the 
estimation  of  V . However,  one  should  recall  thut  the  solcction  of 
the  design  must  uiwuys  be  made  on  the  basis  of  prior  information  that 
is  available  to  the  experimenter. 
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Table  A2.  Data  Simulated  from  the  Lognormal  Response  Curve 
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